The invention relates to gesture recognition in particular gesture recognition by processing of time series of positional inputs received by a two-dimensional (2D) touch sensor, such as a capacitive or resistive touch sensor. The invention may also be applied to one-dimensional (1D) touch sensors, and the principles could also be applied to three-dimensional sensors. It may also be applied to proximity sensors, where no physical contact, i.e. touch, with a sensing surface is involved. The invention can be applied to sensing surfaces operable by a human finger, or a stylus.
1D and 2D capacitive and resistive touch sensors have been in widespread use for many years. Examples include the screens of personal digital assistants (PDAs), MP3 audio player controls, mobile phone keypads and/or displays, and multimedia devices. The touchpad in notebook computers provided in place of a mouse is another form of 2D capacitive touch sensor. 2D sensors are also provided in many domestic appliances, so-called “white goods”, such as ovens and blenders.
Detailed descriptions of 2D capacitive sensors have been given many times, for example in patents and patent applications with the inventor Harald Philipp such as U.S. Pat. No. 2005/0,041,018 A1, U.S. Pat. No. 2007/0,247,443 A1, U.S. Pat. No. 2007/0,257,894 A1, and U.S. Pat. No. 2007/0,279,395 A1, the contents of which are incorporated herein in their entirety.
Other prior art examples of touch screens are as follows.
U.S. Pat. No. 3,593,115 shows a touch element having triangulated shapes for determining object position. However this scheme requires numerous secondary electrode connections as well as two or more layers of construction, increasing construction costs and reducing transparency.
U.S. Pat. No. 5,650,597 shows a 2D sensing method which in its active area requires only one layer but requires large numbers of electrode connections. Resistive strips resolve one axis of position, and the accuracy is dependent on the tolerance of large numbers of resistive strips. This method however does suppress hand shadow effects.
U.S. Pat. No. 6,297,811 describes a touch screen using triangulated wire outline electrode shapes to create field gradients. However this patent suffers from the problem that it is difficult to scale up the screen size, as the number of electrode connections to a sensing circuit is one per triangle. It is desirable to dramatically reduce the number of connections in order to reduce cost and simplify construction. Also it is desirable to use solid shapes rather than wire outlines which are more expensive to construct. This method however does suppress hand shadow effects.
Gesture recognition has also been used for many years in such devices. An early example is character recognition in PDAs, such as the original machines from Palm Inc. Tracking finger motion, and single and double taps on a notebook touchpad is another long used example. More recently, gesture recognition has been incorporated into handheld devices such as the Apple iPhone®. Prior art patent publications on touch screens that involve gesture recognition are also large in number, with significant numbers of publications from Synaptics, Inc. and also more recently Apple Computer, Inc, for example.
U.S. Pat. No. 2007/152,984 A1 assigned to Apple Computer, Inc. discloses a portable communication device with multi-touch input which detects one or more multi-touch contacts and motions and performs one or more operations on an object based on the one or more multi-touch contacts and/or motions.
U.S. Pat. No. 2002/015,024 A1 assigned to University of Delaware discloses simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensor. Segmentation processing extracts shape, position and surface proximity features for each contact and a persistent path tracker is used to detect individual contact touchdown and liftoff. Combinatorial optimization modules associate each contact's path with a particular fingertip, thumb, or palm of either hand on the basis of biomechanical constraints and contact features. Classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.
U.S. Pat. No. 5,825,352 discloses a touch panel which is capable of detecting multiple touches simultaneously. In an xy electrode array, maxima and minima are identified in each of the x and y signals, wherein maxima are designated as finger touches. Peak and valley data in the x and y directions are then interpolated to identify the location of one or more fingers on the sensor array.
U.S. Pat. No. 6,028,271, U.S. Pat. No. 6,414,671 and U.S. Pat. No. 6,750,852 are related patents assigned to Synaptics, Inc. which disclose gesture recognition of an object on a touch-sensor pad and for cursor motion. Tapping, drags, pushes, extended drags and variable drags gestures are recognized by analyzing the position, pressure, and movement of the conductive object on the sensor pad during the time of a suspected gesture, and signals are sent to a host indicating the occurrence of these gestures.
U.S. Pat. No. 2007/176,906 A1 assigned to Synaptics, Inc. discloses a touch sensor having a signal processor adapted to distinguish between three gestures based on different finger motions on the sensing device by providing a workflow with an idle state and three gesture-specific states referred to as first, second and third result states, as illustrated in FIG. 5 of US2007/176906 A1.
Generally, the raw output from the 2D touch sensor will be a time series of x, y coordinates, which are then processed by software, or firmware generated from higher level software, to distinguish the nature of the gesture that has been input. Generally, the raw data is split into contiguous touch segments and then processed to determine what if any gestures can be deduced. The processing of the raw data to identify the gestures may be carried out in the same chip as generates the raw data, or the raw data may be exported to an external chip, for example by transmission over a communication bus to the device's central processing unit (CPU). The former approach is preferred by Synaptics, the latter by Apple as exemplified by U.S. Pat. No. 2006/0,066,582 A1.
Most of the patent literature is unspecific about how the raw time series data are converted into gestures. The straightforward approach is to write appropriate high level code, for example in C or another suitable programming language, in which the interpretation of the time series data is analyzed using conditional statements, such as if . . . then . . . else.
However, it is difficult to reliably and efficiently add code to identify a new gesture into an existing block of code for distinguishing between a significant number of gestures, for example at least 3 or 4, perhaps 10 to 20. Testing of the code is a particular difficulty. This is because in general at any intermediate point in a time series of x,y,t data the input may relate to a plurality of possible gestures, thereby making the coding for recognizing one gesture generally dependent on or linked to the coding for recognizing another gesture.